Enhancing the Reasoning Capabilities of Natural Language Inference Models with Attention Mechanisms and External Knowledge

Natural Language Inference (NLI) is fundamental to natural language understanding. The task summarises the natural language understanding capabilities within a simple formulation of determining whether a natural language hypothesis can be inferred from a given natural language premise. NLI requires an inference system to address the full complexity of linguistic as well as real-world commonsense knowledge and, hence, the inferencing and reasoning capabilities of an NLI system are utilised in other complex language applications such as summarisation and machine comprehension. Consequently, NLI has received significant recent attention from both academia and industry. Despite extensive research, contemporary neural NLI models face challenges arising from the sole reliance on training data to comprehend all the linguistic and real-world commonsense knowledge. Further, different attention mechanisms, crucial to the success of neural NLI models, present the prospects of better utilisation when employed in combination. In addition, the NLI research field lacks a coherent set of guidelines for the application of one of the most crucial regularisation hyper-parameters in the RNN-based NLI models -- dropout. In this thesis, we present neural models capable of leveraging the attention mechanisms and the models that utilise external knowledge to reason about inference. First, a combined attention model to leverage different attention mechanisms is proposed. Experimentation demonstrates that the proposed model is capable of better modelling the semantics of long and complex sentences. Second, to address the limitation of the sole reliance on the training data, two novel neural frameworks utilising real-world commonsense and domain-specific external knowledge are introduced. Employing the rule-based external knowledge retrieval from the knowledge graphs, the first model takes advantage of the convolutional encoders and factorised bilinear pooling to augment the reasoning capabilities of the state-of-the-art NLI models. Utilising the significant advances in the research of contextual word representations, the second model, addresses the existing crucial challenges of external knowledge retrieval, learning the encoding of the retrieved knowledge and the fusion of the learned encodings to the NLI representations, in unique ways. Experimentation demonstrates the efficacy and superiority of the proposed models over previous state-of-the-art approaches. Third, for the limitation on dropout investigations, formulated on exhaustive evaluation, analysis and validation on the proposed RNN-based NLI models, a coherent set of guidelines is introduced

ExBERT: An External Knowledge Enhanced BERT for Natural Language Inference

Neural language representation models such as BERT, pretrained on large-scale unstructured corpora lack explicit grounding to real-world commonsense knowledge and are often unable to remember facts required for reasoning and inference. Natural Language Inference (NLI) is a challenging reasoning task that relies on common human understanding of language and real-world commonsense knowledge. We introduce a new model for NLI called External Knowledge Enhanced BERT (ExBERT), to enrich the contextual representation with realworld commonsense knowledge from external knowledge sources and enhance BERT’s language understanding and reasoning capabilities. ExBERT takes full advantage of contextual word representations obtained from BERT and employs them to retrieve relevant external knowledge from knowledge graphs and to encode the retrieved external knowledge. Our model adaptively incorporates the external knowledge context required for reasoning over the inputs. Extensive experiments on the challenging SciTail and SNLI benchmarks demonstrate the effectiveness of ExBERT: in comparison to the previous state-of-the-art, we obtain an accuracy of 95.9% on SciTail and 91.5% on SNLI

An Exploration of Dropout with RNNs for Natural Language Inference

Dropout is a crucial regularization technique for the Recurrent Neural Network (RNN) models of Natural Language Inference (NLI). However, dropout has not been evaluated for the effectiveness at different layers and dropout rates in NLI models. In this paper, we propose a novel RNN model for NLI and empirically evaluate the effect of applying dropout at different layers in the model. We also investigate the impact of varying dropout rates at these layers. Our empirical evaluation on a large (Stanford Natural Language Inference (SNLI)) and a small (SciTail) dataset suggest that dropout at each feed-forward connection severely affects the model accuracy at increasing dropout rates. We also show that regularizing the embedding layer is efficient for SNLI whereas regularizing the recurrent layer improves the accuracy for SciTail. Our model achieved an accuracy 86.14% on the SNLI dataset and 77.05% on SciTail

Modelling commonsense properties using pre-trained bi-encoders

Grasping the commonsense properties of everyday concepts is an important prerequisite to language understanding. While contextualised language models are reportedly capable of predicting such commonsense properties with human-level accuracy, we argue that such results have been inflated because of the high similarity between training and test concepts. This means that models which capture concept similarity can perform well, even if they do not capture any knowledge of the commonsense properties themselves. In settings where there is no overlap between the properties that are considered during training and testing, we find that the empirical performance of standard language models drops dramatically. To address this, we study the possibility of fine-tuning language models to explicitly model concepts and their properties. In particular, we train separate concept and property encoders on two types of readily available data: extracted hyponym-hypernym pairs and generic sentences. Our experimental results show that the resulting encoders allow us to predict commonsense properties with much higher accuracy than is possible by directly fine-tuning language models. We also present experimental results for the related task of unsupervised hypernym discovery

What do Deck Chairs and Sun Hats Have in Common? Uncovering Shared Properties in Large Concept Vocabularies

Concepts play a central role in many applications. This includes settings where concepts have to be modelled in the absence of sentence context. Previous work has therefore focused on distilling decontextualised concept embeddings from language models. But concepts can be modelled from different perspectives, whereas concept embeddings typically mostly capture taxonomic structure. To address this issue, we propose a strategy for identifying what different concepts, from a potentially large concept vocabulary, have in common with others. We then represent concepts in terms of the properties they share with the other concepts. To demonstrate the practical usefulness of this way of modelling concepts, we consider the task of ultra-fine entity typing, which is a challenging multi-label classification problem. We show that by augmenting the label set with shared properties, we can improve the performance of the state-of-the-art models for this task.Comment: Accepted for EMNLP 202

To evaluate and compare the efficacy of alcoholic and aqueous extract of Lagenaria siceraria in high fat diet model in wistar rats

Background: Obesity is not only affecting the affluent society but also affecting developing countries like India. The incidence of obesity is rapidly increasing throughout the world. However, the current anti-obesity drugs have numerous limitations.Methods: The obesity was induced in male wistar rats by giving high-fat diet over 12 weeks. The variables assessed were body weight, abdominal girth, blood triglyceride level, liver weight and fat mass and histopathology of liver. Aqueous and alcoholic extracts of Lagenaria siceraria (200mg/kg and 400mg/kg Doses) were compared to orlistat (treatment control) and high-fat diet group (disease control) for different variables.Results: Alcoholic and aqueous extracts high dose (400mg/kg) of Lagenaria siceraria significantly reduced total body weight (p<0.05), abdominal girth (p <0.05) at week 10 and 12 compared to high fat diet group. Alcoholic extract (400mg/kg) significantly reduced total blood triglyceride level (p <0.05) and total liver weight (p <0.05) compared to high-fat diet group. None of the study drugs reduced % liver weight. Alcoholic extract high dose (p <0.05) has shown improvement in histopathological score. Both aqueous and alcoholic extracts have shown reduced fat mass compared to high-fat diet group.Conclusions: The alcoholic extract (400mg/kg) of Lagenaria siceraria alleviated high fat diet induced obesity and dyslipidemic changes in rats. The alcoholic extract of Lagenaria siceraria is having better anti-obesity potential than aqueous extract

deepQuest-py: large and distilled models for quality estimation

© (2021) The Authors. Published by Association for Computational Linguistics. This is an open access article available under a Creative Commons licence. The published version can be accessed at the following link on the publisher’s website: https://aclanthology.org/2021.emnlp-demo.42/We introduce deepQuest-py, a framework for training and evaluation of large and lightweight models for Quality Estimation (QE). deepQuest-py provides access to (1) state-ofthe-art models based on pre-trained Transformers for sentence-level and word-level QE; (2) light-weight and efficient sentence-level models implemented via knowledge distillation; and (3) a web interface for testing models and visualising their predictions. deepQuestpy is available at https://github.com/ sheffieldnlp/deepQuest-py under a CC BY-NC-SA licence

Preparation and evaluation of mouth dissolving tablets of meloxicam

The aim of the present study was to develop evaluate mouth dissolving tablet of meloxicam. Drug delivery systems became sophisticated as pharmaceutical scientists acquire a better understanding of the physicochemical and biochemical parameters pertinent to their performance. Over the past three decades, mouth dissolving or orally disintegrating tablets have gained considerable attention as a preferred alternative to conventional tablets due to better patient compliance. The most preferrable route of drug administration (e.g. oral) is limited to drug candidate that show poor permeability across the gastric mucosa and those, which are sparingly soluble. A large majority of the new chemical entities and many new existing drug molecules are poorly soluble, thereby limiting their potential uses and increasing the difficulty of formulating bioavailable drug products,so lastlly the purpose of this study was to grow mouth dissolve tablets of Meloxicam. Meloxicam is a newer selective COX-1 inhibitor. These tablets were prepared by wet granulation procedure. The tablets were evaluated for % friability, wetting time and disintegration time. Sublimation of camphor from tablets resulted in better tablets as compared to the tablets prepared from granules that were exposing to vacuum. The systematic formulation approach helped in understanding the effect of formulation processing variables.Keywords: Mouth dissolving tablet; Maloxicam; Bioavailability; NSAI

Preparation and evaluation of mouth dissolving tablets of meloxicam

The aim of the present study was to develop evaluate mouth dissolving tablet of meloxicam. Drug delivery systems became sophisticated as pharmaceutical scientists acquire a better understanding of the physicochemical and biochemical parameters pertinent to their performance. Over the past three decades, mouth dissolving or orally disintegrating tablets have gained considerable attention as a preferred alternative to conventional tablets due to better patient compliance. The most preferrable route of drug administration (e.g. oral) is limited to drug candidate that show poor permeability across the gastric mucosa and those, which are sparingly soluble. A large majority of the new chemical entities and many new existing drug molecules are poorly soluble, thereby limiting their potential uses and increasing the difficulty of formulating bioavailable drug products,so lastlly the purpose of this study was to grow mouth dissolve tablets of Meloxicam. Meloxicam is a newer selective COX-1 inhibitor. These tablets were prepared by wet granulation procedure. The tablets were evaluated for % friability, wetting time and disintegration time. Sublimation of camphor from tablets resulted in better tablets as compared to the tablets prepared from granules that were exposing to vacuum. The systematic formulation approach helped in understanding the effect of formulation processing variables.Keywords: Mouth dissolving tablet; Maloxicam; Bioavailability; NSAI